Skip to content

fix(input-handler): bound URL, zip, and git ingest paths#164

Open
rcha0s wants to merge 1 commit into
NVIDIA:mainfrom
rcha0s:feat/bound-ingest-layer
Open

fix(input-handler): bound URL, zip, and git ingest paths#164
rcha0s wants to merge 1 commit into
NVIDIA:mainfrom
rcha0s:feat/bound-ingest-layer

Conversation

@rcha0s

@rcha0s rcha0s commented Jun 23, 2026

Copy link
Copy Markdown

Summary

Closes #21. Closes #131.

The per-file analysis cap (MAX_FILE_BYTES, 1 MB) sits downstream of InputHandler.resolve(), which pulls scan targets from URLs, zips, or git clones with no size budget of its own. The per-file cap can therefore be defeated upstream:

  • a multi-GB URL is a memory DoS (client.get(url).content buffers the whole body),
  • a zip bomb fills the disk before extraction is gated (extractall with no size pre-check),
  • a large clone lands on disk regardless of the per-file analysis budget (no post-clone size check).

@wernerkasselman-au flagged this in the review of PR #19 and explicitly deferred it ("the whole ingest layer in input_handler.py runs before build_context ever sees a file, and it is unbounded"). PR #19 stayed scoped to the per-file work; this PR addresses the deferred ingest-layer hardening.

Changes

New constants

Constant Value Purpose
INGEST_MAX_BYTES 100 MiB Cap on streamed URL downloads, total uncompressed size of zip archives, and post-clone disk usage
INGEST_MAX_ZIP_MEMBERS 10,000 Cap on the number of entries in a single zip — defends the "many tiny files" zip-bomb variant where each entry is small but the count itself exhausts the FS

Sized above the existing 1 MB per-file analysis cap so legitimate multi-file skills aren't blocked at ingest, but tight enough to bound a memory / disk DoS.

_download_file — streamed with hard cap

  • Replaces client.get(url).content (full-body buffering) with client.stream("GET", url) + a chunked iterator.
  • Up-front Content-Length check — if the server provides a Content-Length larger than INGEST_MAX_BYTES, abort before reading any body bytes.
  • Streamed byte counter is authoritative — if the header is missing, malformed, or wrong, the running count of bytes received via iter_bytes() aborts the read as soon as it crosses the cap.

_extract_zip — zip-bomb defended

  • Sums ZipInfo.file_size across all members and checks the member count before calling extractall. Classic zip bombs (small archive, huge declared uncompressed size) are rejected without materialising any of the bomb on disk.
  • Note: this is complementary to PR fix(input-handler): validate URLs against SSRF and add zip-slip protection #159 (zip-slip path traversal); that PR catches members with .. in their names, this one catches the size/count attack class.

_clone_git — post-clone size check

  • Walks the cloned tree after the existing 60s timeout completes and rejects + cleans up (shutil.rmtree) if total size exceeds INGEST_MAX_BYTES.
  • Symlinks are skipped (p.is_symlink() check) to avoid runaway counts on malicious /dev/zero-style links.

Error semantics

All limit breaches raise IngestLimitExceededError — a subclass of ValueError so existing callers that catch ValueError from InputHandler.resolve() continue to work. Error messages include both the limit name and the observed size for clarity.

Test plan

  • 8 new unit tests in tests/unit/test_input_handler_bounds.py:
    • Under-cap downloads / zips / clones succeed unchanged.
    • Oversized Content-Length header rejected before body read (uses httpx.MockTransport with a forged header).
    • Streamed body overflow rejected when header is missing (generator-backed stream so httpx can't pre-compute Content-Length).
    • Declared-uncompressed-oversize zip rejected without extraction (forged central-directory file_size field).
    • Too many members rejected (10,001 entries).
    • Oversize clone rejected and the partial clone dir cleaned up.
  • Existing 6 input-handler tests continue to pass.
  • Full suite: 728 passed, 12 skipped.
  • ruff check clean.
  • ruff format --check clean.
  • DCO sign-off on the commit.

README

Added a short "Size limits" subsection under Basic Usage documenting both caps and their relationship to the per-file analysis cap (per the maintainer's deferred-work request: "Document in the README that 50 MiB is a per-file analysis limit, not an ingest limit").

Notes / follow-up

  • The constants are module-level rather than CLI-configurable. If you'd prefer --ingest-max-bytes / --ingest-max-zip-members flags or env vars (SKILLSPECTOR_INGEST_MAX_BYTES), happy to add — wanted to keep this PR scoped.
  • PR fix(input-handler): validate URLs against SSRF and add zip-slip protection #159 covers SSRF + zip-slip path traversal on the same code paths; that work is orthogonal to size bounds and the two PRs are independently mergeable.

@rcha0s rcha0s force-pushed the feat/bound-ingest-layer branch from 56cca8f to cf31832 Compare June 23, 2026 02:31
@rcha0s

rcha0s commented Jun 23, 2026

Copy link
Copy Markdown
Author

Force-pushed cf31832 after a security self-review surfaced a real issue: the original v1 streamed bytes through a Python list (chunks.append(chunk)) and the cap check fired after the append, so a 100 MiB attack could still load ~100 MiB into memory before being rejected — and the doubled allocation from b"".join(chunks) made the effective peak ~200 MiB. Not the original "unbounded DoS" the PR was meant to fix, but a meaningful regression of the security claim.

v2 changes

  • _download_file streams directly to a temp file, not an in-memory list. The body is never held as a single bytes object. Peak memory is now bounded by _DOWNLOAD_CHUNK_BYTES (64 KiB), not INGEST_MAX_BYTES.
  • Partial file is removed on breach. If the byte counter trips mid-stream, the partial download is unlink'd before the exception propagates. Without this, an attacker who ships exactly INGEST_MAX_BYTES + 1 bytes could fill the temp dir to the cap on every request. Also covered for HTTP errors (transient failures don't leave partial files behind either).
  • Same change covers the zip download path. A downloaded zip is now rename'd into place from the partial file instead of being written from an in-memory buffer, so the same memory-pressure ceiling holds whether the URL points at a markdown file or a zip.

Two new tests

  • test_streamed_overflow_leaves_no_partial_file_on_disk — verifies the partial file does not survive a mid-stream breach.
  • test_download_streams_to_disk_not_memory — verifies a legitimate 5 MiB download lands on disk at the expected size with the sentinel partial-download path cleaned up. Can't directly measure memory in a unit test, but the file-shape assertion is a proxy: with the v1 implementation, the file would be written from a b"".join(chunks) allocation, which the test would still pass but the memory pressure would be visible at scale; with v2 the file is written incrementally.

Other security-review findings — disposition

Finding Severity Disposition
Memory accumulation in download (this fix) MEDIUM Fixed in v2
Git clone fully lands on disk before measurement LOW-MED Documented limitation. Mitigating would need git clone --filter=blob:limit=..., which adds complexity for a small win when the runner's /tmp is already constrained. Open to adding if you'd prefer.
_is_git_url substring match ("github.com" in netloc) is a confused-deputy hazard LOW for this PR (pre-existing), HIGH at repo level Out of scope here. Will flag separately on PR #159 since that one's already in the SSRF / host-validation space.
_make_bomb_zip test helper is fragile to multi-member extensions LOW (test brittleness, not security) Helper already has a spec ref + assert; deferring beyond that to keep scope tight.
Symlinks-to-files-inside-the-clone undercount in _directory_size_bytes LOW The attack requires the symlink target to already be on the filesystem outside the clone (the clone walks a fresh tmpdir), so the practical exploit surface is narrow. Documented in the docstring already.

Verification

  • 10 unit tests in tests/unit/test_input_handler_bounds.py (8 original + 2 new).
  • Full suite: 730 passed, 12 skipped.
  • ruff check + ruff format --check: clean.
  • Commit message updated; DCO sign-off preserved.

Ready for re-review.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Approve — excellent, security-aware DoS hardening that fails closed.

What's good

  • Streamed chunked download with an up-front Content-Length check and a streamed byte counter (src/skillspector/input_handler.py, ~L145-170), so an oversized/mislabeled body aborts before buffering in memory; partial file is cleaned up on breach (~L172-182).
  • Zip: member-count cap + summed-uncompressed-size cap before extractall (~L212-224); post-clone directory-size check with cleanup for git (~L96-111); _directory_size_bytes is symlink-safe (~L234-249).
  • IngestLimitExceededError(ValueError) keeps back-compat with existing ValueError callers. Clear, documented constants and README note.

Non-blocking (security-relevant)

  • The zip check sums ZipInfo.file_size (the declared central-directory uncompressed size, ~L218). A crafted bomb that under-declares uncompressed size could still inflate on extractall. For full protection, consider per-member streamed extraction with a running on-disk byte counter that aborts mid-extract. (The forged-central-directory test exercises the declared-size path, not the under-declared case.)

Coordination

Overlaps #159 (this PR lacks the SSRF host allowlist #159 adds) and #163. Reconcile on merge so the final _download_file/_extract_zip has both SSRF allowlisting and ingest size bounds.

Tests

Thorough: MockTransport download caps (header vs streamed), partial-file cleanup, forged central-directory zip bomb, member-count bomb, oversize-clone cleanup. LGTM.

@rng1995

rng1995 commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Please resolve the merge conflicts

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Automated SkillSpector Review]

Summary

This PR bounds the three ingest paths in InputHandler (URL download, zip extraction, git clone) with INGEST_MAX_BYTES (100 MiB) and INGEST_MAX_ZIP_MEMBERS (10,000), raising a fail-closed IngestLimitExceededError (a ValueError subclass). The core design is good: streamed chunked downloads with an up-front Content-Length check plus an authoritative streamed byte counter, partial-file cleanup on breach, zip declared-uncompressed-size and member-count pre-checks before extractall, and a post-clone disk-size check with cleanup. The 8 new tests are thorough and include the partial-file-cleanup and forged-Content-Length cases.

However, the branch is based on a main that predates the SSRF/zip-slip hardening that has since merged, and it cannot be merged or safely evaluated as-is. (Note: this supersedes the earlier automated approval on this PR, which predated the conflicting security work on main.)

Blockers

  1. Merge conflict (mergeable_state: dirty). The maintainer already asked for conflicts to be resolved on 2026-06-23 and the branch has not been updated since. Main now contains c4aab55 (URL host allowlist + SSRF validation + zip-slip member check), f5305b9 (follow_redirects=False to close an SSRF redirect bypass), and 680cc0f (scp-style git URL host validation), all touching the same functions this PR rewrites.
  2. Conflict resolution is security-sensitive. The PR's _download_file still constructs httpx.Client(follow_redirects=True, ...) and has no _validate_url_host() calls, and its _extract_zip lacks main's zip-slip member-path loop. A rebase that takes this PR's side of those hunks would silently reintroduce the SSRF redirect bypass, drop the download/git host allowlist, and drop zip-slip protection. The rebase must preserve follow_redirects=False, both _validate_url_host() call sites, and the zip-slip loop alongside the new size checks.
  3. New tests fail against current main. Every download-path test resolves https://example.com/..., but main's _download_file rejects any host not in ALLOWED_DOWNLOAD_HOSTS (and _is_private_ip does a real DNS lookup) before the mocked transport is ever reached. After rebase these tests need an allowlisted host (e.g. raw.githubusercontent.com) and should monkeypatch the host/IP validation to avoid network-dependent DNS in unit tests. The claimed "728 passed" run was against the stale base.

Non-blocking

  • _directory_size_bytes docstring says "Path.is_file() returns True only for regular files" — incorrect; is_file() follows symlinks. The code is only correct because of the explicit not p.is_symlink() guard; fix the docstring so a future refactor doesn't remove the guard based on the stated rationale.
  • The clone size check is post-hoc: the full tree lands on disk before rejection, so within the 60s timeout an attacker can still consume disk transiently. Worth a code comment documenting this residual window (the PR body acknowledges it; the code should too).
  • The size walk counts .git/ objects toward the cap, so a legitimate repo with a working tree just under 100 MiB can still be rejected; consider excluding .git or documenting the behavior.
  • Zip declared-size bound is sound for stdlib extraction (CPython's zipfile truncates output at the declared file_size), so the forged-central-directory test is meaningful — nice.

Please rebase onto current main, reconcile with the merged SSRF/zip-slip work, fix the tests to pass against the allowlist, and re-run the full suite on the rebased branch.

Comment thread src/skillspector/input_handler.py Outdated
download_path = temp_dir / "_download.partial"
content_type = ""
try:
with httpx.Client(follow_redirects=True, timeout=30) as client:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch predates main's SSRF hardening: main now uses follow_redirects=False (commit f5305b9, closing a redirect-based allowlist bypass) and calls _validate_url_host(url, ALLOWED_DOWNLOAD_HOSTS) at the top of _download_file (c4aab55). When resolving the merge conflicts, both must be preserved — keeping follow_redirects=True here would reintroduce the SSRF bypass.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the rebase (e5e5a21). Both preserved:

  • httpx.Client(follow_redirects=False, timeout=30) — see _download_file in the rebased input_handler.py (the streamed-download wrapper is the only change to that constructor).
  • self._validate_url_host(url, ALLOWED_DOWNLOAD_HOSTS) is the first line of _download_file after the docstring, so the allowlist + private-IP check runs before any bytes are read.

The same pattern applies on _clone_git: self._validate_url_host(url, ALLOWED_GIT_HOSTS) runs before subprocess.run.

extract_dir.mkdir(exist_ok=True)
try:
with zipfile.ZipFile(zip_path, "r") as zf:
infos = zf.infolist()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main's _extract_zip now also iterates zf.namelist() and rejects members that resolve outside extract_dir (zip-slip, c4aab55). The rebased version needs both that loop and these size/member-count checks; taking this PR's side of the conflict would drop zip-slip protection.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the rebase. _extract_zip now runs, in order:

  1. The INGEST_MAX_ZIP_MEMBERS check.
  2. The sum(info.file_size) vs INGEST_MAX_BYTES check.
  3. A zf.namelist() loop that resolves each member against extract_dir.resolve() and raises ValueError("...zip-slip...") if the resolved path is not inside the extraction root.
  4. zf.extractall(extract_dir).

So the size caps fire first (cheap, protects against bombs before we even touch member names), then zip-slip runs, then extract. Nothing was dropped from main's protections.

Comment thread src/skillspector/input_handler.py Outdated
def _directory_size_bytes(path: Path) -> int:
"""Return the total size of all regular files under *path*, in bytes.

Symlinks are not followed (``Path.is_file()`` returns ``True`` only

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docstring is incorrect: Path.is_file() follows symlinks and returns True for a symlink to a regular file. The code below is only safe because of the explicit not p.is_symlink() guard — please fix the rationale here so the guard isn't removed later on the strength of this comment.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. _directory_size_bytes docstring rewritten to state exactly what you flagged:

Symlinks are explicitly skipped via Path.is_symlink() — note that Path.is_file() follows symlinks and would otherwise return True for a symlink pointing at a regular file, so the not p.is_symlink() guard is load-bearing and must not be removed. This is what prevents a malicious symlink to /dev/zero (or any large file outside the walked tree) from inflating the count.

So the rationale in the comment now argues for the guard rather than against it — a future refactor that reads this docstring and considers dropping the check will be pushed back toward keeping it.

# Post-clone size check: a successful --depth 1 clone may still
# land an arbitrarily large tree on disk. Reject (and clean up)
# if it exceeds the ingest budget.
total = _directory_size_bytes(clone_dir)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this check is post-hoc: the entire clone already landed on disk before it runs, so disk is transiently consumable up to whatever fits in the 60s timeout. Acceptable as a bound, but worth a comment documenting the residual window. Also, .git/ objects count toward the cap, so a legitimate repo with a working tree just under 100 MiB may be rejected.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented as requested. Added an in-code comment right above the size check in _clone_git:

# Post-clone size check: a successful --depth 1 clone may still
# land an arbitrarily large tree on disk before we can measure
# it, so this is a fail-closed cap rather than a hard prefilter.
# Residual window: within the 60s clone timeout an attacker can
# transiently consume up to whatever the network + disk let
# through before this check runs; bounded by the timeout, but
# not zero.  `.git/` objects are counted toward the cap, so a
# legitimate repo with a working tree just under `INGEST_MAX_BYTES`
# can still be rejected once packfiles are added.

Deliberately not fixing either issue in this PR:

  • The residual disk window would require git clone --filter=blob:limit=… (partial clone) plus streaming the pack, which is a much larger change and mainly buys tighter bounding on the same disk that the 60s timeout already caps. Documented, not addressed.
  • .git/ counting toward the cap is arguably correct — an attacker who ships a bloated packfile with a small working tree would otherwise slip through. Excluding .git/ would need a separate size accounting pass. Also documented for future callers who hit it on legitimate repos.

Happy to spin either out as follow-up issues if you'd rather have them tracked.

Comment thread tests/unit/test_input_handler_bounds.py Outdated

h = InputHandler()
try:
resolved, source_type = h.resolve("https://example.com/skill.md")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

example.com is not in ALLOWED_DOWNLOAD_HOSTS on current main, so after rebase _validate_url_host raises ValueError before the MockTransport is reached — this and the other download tests will fail. Use an allowlisted host (e.g. raw.githubusercontent.com) and monkeypatch _is_private_ip (it performs a real DNS lookup) so the unit tests stay hermetic.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both fixed:

  1. Host swap. All download-path tests now use https://raw.githubusercontent.com/... (in ALLOWED_DOWNLOAD_HOSTS) instead of example.com. sed-style change — six tests updated: test_under_cap_downloads_succeed, test_content_length_header_rejected_before_body_read, test_streamed_body_overflow_rejected_when_header_missing, test_streamed_overflow_leaves_no_partial_file_on_disk, test_download_streams_to_disk_not_memory.
  2. DNS-free unit tests. _patch_httpx_client now also monkeypatches ih._is_private_ip to a lambda host: False, so _validate_url_host doesn't call socket.getaddrinfo in the unit-test path. Added a matching _stub_private_ip_check(monkeypatch) helper for the two TestGitCloneBound tests.

Also had to touch two pre-existing tests in tests/unit/test_input_handler_ssrf.py (test_raw_githubusercontent_allowed, test_download_does_not_follow_redirects) because _download_file now uses client.stream("GET", url) as a context manager instead of client.get(url) — the mocks needed the nested __enter__ + iter_bytes shape. Same behavioral coverage, updated fixtures.

Full suite result on the rebased branch: 1320 passed, 14 skipped, 6 xfailed. ruff check + ruff format --check clean.

Closes NVIDIA#21. Closes NVIDIA#131.

The per-file analysis cap (MAX_FILE_BYTES, 1 MB) sits downstream of
InputHandler.resolve(), which pulls scan targets from URLs, zips, or
git clones with no size budget of its own.  As a result, the per-file
cap can be defeated upstream: a multi-GB URL is a memory DoS, a zip
bomb fills the disk before extraction is gated, and a large clone
lands on disk regardless of the per-file analysis budget.  PR NVIDIA#19
explicitly deferred this; this PR addresses the deferred work.

Adds two ingest budgets enforced at every remote/archive ingest path:

- INGEST_MAX_BYTES (100 MiB): caps streamed URL downloads, total
  uncompressed size of zip archives, and post-clone disk usage of
  Git repos.
- INGEST_MAX_ZIP_MEMBERS (10,000): caps the number of entries in a
  single zip (defends against the "many tiny files" zip-bomb variant).

Per ingest path:

- _download_file streams the body in 64 KiB chunks directly to a temp
  file inside the existing session temp dir, with a running byte
  counter.  The cap check fires before each chunk is written, so the
  response body is never accumulated in memory.  Content-Length is
  checked up-front when the server provides it (aborts before reading
  any body bytes); the streamed counter is authoritative if the
  header is missing or malformed.  A breach mid-stream removes the
  partial file before the exception propagates, so an attacker who
  ships exactly INGEST_MAX_BYTES + 1 bytes cannot fill the temp dir.
- _extract_zip sums ZipInfo.file_size across all members and checks
  the member count *before* calling extractall, so classic zip bombs
  (small archive, huge declared uncompressed size) are rejected
  without materialising any of the bomb on disk.
- _clone_git measures the cloned tree's on-disk size after the
  existing 60s timeout completes and rejects + cleans up if it
  exceeds the cap.  Symlinks are skipped to avoid runaway counts on
  malicious /dev/zero-style links.

All limit breaches raise IngestLimitExceededError (subclass of
ValueError so existing callers that catch ValueError keep working)
with a clear message including the limit and the observed size.

Adds 10 unit tests covering: under-cap success paths for URL/zip/clone;
oversized Content-Length rejected before body read; chunked overflow
caught by the streamed counter; *the partial download file is removed
when a breach fires mid-stream*; *streaming to disk produces a file of
the expected size with no intermediate in-memory concatenation*;
declared-uncompressed-oversize zip rejected without extraction;
member-count zip-bomb rejected; oversize clone rejected and cleaned
up.  Full suite: 730 passed, 12 skipped.

README documents both caps and their relationship to the per-file
analysis cap.

Signed-off-by: Rohan Isawe <[email protected]>
@rcha0s rcha0s force-pushed the feat/bound-ingest-layer branch from cf31832 to e5e5a21 Compare July 14, 2026 19:32
@rcha0s

rcha0s commented Jul 14, 2026

Copy link
Copy Markdown
Author

Pushed rebase e5e5a21. Every blocker and non-blocker from the 2026-07-14 review is addressed — inline replies on each thread with the specific fix. TL;DR:

# Finding Disposition
1 Merge conflict (mergeable_state: dirty) Rebased onto current main. mergeable: MERGEABLE.
2 Rebase must preserve SSRF hardening _validate_url_host calls, follow_redirects=False, and the zip-slip loop all kept alongside the new size checks.
3 Download tests use non-allowlisted example.com Swapped to raw.githubusercontent.com; _is_private_ip monkeypatched to keep tests hermetic.
4 _directory_size_bytes docstring wrong about is_file() symlink behavior Docstring rewritten to say the not p.is_symlink() guard is load-bearing.
5 Clone size check is post-hoc + .git/ counted In-code comment added. Not addressing the residual window / .git/ accounting in this PR — happy to open follow-up issues.

Verification on the rebased branch:

  • Full suite: 1320 passed, 14 skipped, 6 xfailed.
  • ruff check + ruff format --check: clean.

Ready for re-review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants